Combining Linguistics with statistics for multiword term extraction: a fruitfull association?

The acquisition of multiword terms from large text collections is a fundamental issue in the context of Information Retrieval. Indeed, their identification leads to improvements in the indexing process and allows guiding the user in his search for information. In this paper, we present an original methodology that allows extracting multiword terms by either (1) exclusively considering statistical word regularities or by (2) combining word statistics with endogenously acquired linguistic information. For that purpose, we conjugate a new association measure called the Mutual Expectation with a new acquisition process called the LocalMaxs. On one hand, the Mutual Expectation, based on the concept of Normalised Expectation, evaluates the degree of cohesiveness that links together all the textual units contained in an n-gram (i.e. ∀n, n ≥ 2). On the other hand, the LocalMaxs retrieves the candidate terms from the set of all the valued n-grams by evidencing local maxima of association measure values. Finally, we compare the results obtained by applying the methodology over a raw Portuguese text with the results reached by combining word statistics with linguistic information endogenously acquired from the same corpus previously tagged.